System, method and storage medium for providing fault detection and correction in a memory subsystem

ABSTRACT

A memory subsystem with a memory bus and a memory assembly. The memory bus includes multiple bitlanes. The memory assembly is in communication with the memory bus and includes instructions for receiving an error code correction (ECC) word in multiple packets via the memory bus. The ECC word includes data bits and ECC bits arranged into multiple multi-bit ECC symbols. Each of the ECC symbols is associated with one of the bitlanes on the memory bus. The memory assembly also includes instructions for utilizing one of the ECC symbols to perform error detection and correction for the bits in the ECC word received via the bitlane associated with the ECC symbol.

BACKGROUND OF THE INVENTION

The invention relates to a memory subsystem and in particular, toproviding fault detection and correction in a memory subsystem.

Computer memory subsystems have evolved over the years, but continue toretain many consistent attributes. Computer memory subsystems from theearly 1980's, such as the one disclosed in U.S. Pat. No. 4,475,194 toLaVallee et al., of common assignment herewith, included a memorycontroller, a memory assembly (contemporarily called a basic storagemodule (BSM) by the inventors) with array devices, buffers, terminatorsand ancillary timing and control functions, as well as severalpoint-to-point busses to permit each memory assembly to communicate withthe memory controller via its own point-to-point address and data bus.FIG. 1 depicts an example of this early 1980 computer memory subsystemwith two BSMs, a memory controller, a maintenance console, andpoint-to-point address and data busses connecting the BSMs and thememory controller.

FIG. 2, from U.S. Pat. No. 5,513,135 to Dell et al., of commonassignment herewith, depicts an early synchronous memory module, whichincludes synchronous dynamic random access memories (DRAMs) 8, bufferdevices 12, an optimized pinout, an interconnect and a capacitivedecoupling method to facilitate operation. The patent also describes theuse of clock re-drive on the module, using such devices as phase lockloops (PLLs).

FIG. 3, from U.S. Pat. No. 6,510,100 to Grundon et al., of commonassignment herewith, depicts a simplified diagram and description of amemory system 10 that includes up to four registered dual inline memorymodules (DIMMs) 40 on a traditional multi-drop stub bus channel. Thesubsystem includes a memory controller 20, an external clock buffer 30,registered DIMMs 40, an address bus 50, a control bus 60 and a data bus70 with terminators 95 on the address bus 50 and data bus 70.

FIG. 4 depicts a 1990's memory subsystem which evolved from thestructure in FIG. 1 and includes a memory controller 402, one or morehigh speed point-to-point channels 404, each connected to a bus-to-busconverter chip 406, and each having a synchronous memory interface 408that enables connection to one or more registered DIMMs 410. In thisimplementation, the high speed, point-to-point channel 404 operated attwice the DRAM data rate, allowing the bus-to-bus converter chip 406 tooperate one or two registered DIMM memory channels at the full DRAM datarate. Each registered DIMM included a PLL, registers, DRAMs, anelectrically erasable programmable read-only memory (EEPROM) andterminators, in addition to other passive components.

As shown in FIG. 5, memory subsystems were often constructed with amemory controller connected either to a single memory module, or to twoor more memory modules interconnected on a ‘stub’ bus. FIG. 5 is asimplified example of a multi-drop stub bus memory structure, similar tothe one shown in FIG. 3. This structure offers a reasonable tradeoffbetween cost, performance, reliability and upgrade capability, but hasinherent limits on the number of modules that may be attached to thestub bus. The limit on the number of modules that may be attached to thestub bus is directly related to the data rate of the informationtransferred over the bus. As data rates increase, the number and lengthof the stubs must be reduced to ensure robust memory operation.Increasing the speed of the bus generally results in a reduction inmodules on the bus with the optimal electrical interface being one inwhich a single module is directly connected to a single controller, or apoint-to-point interface with few, if any, stubs that will result inreflections and impedance discontinuities. As most memory modules aresixty-four or seventy-two bits in data width, this structure alsorequires a large number of pins to transfer address, command, and data.One hundred and twenty pins are identified in FIG. 5 as being arepresentative pincount.

FIG. 6, from U.S. Pat. No. 4,723,120 to Petty, of common assignmentherewith, is related to the application of a daisy chain structure in amultipoint communication structure that would otherwise require multipleports, each connected via point-to-point interfaces to separate devices.By adopting a daisy chain structure, the controlling station can beproduced with fewer ports (or channels), and each device on the channelcan utilize standard upstream and downstream protocols, independent oftheir location in the daisy chain structure.

FIG. 7 represents a daisy chained memory bus, implemented consistentwith the teachings in U.S. Pat. No. 4,723,120. A memory controller 111is connected to a memory bus 315, which further connects to a module 310a. The information on memory bus 315 is re-driven by the buffer onmodule 310 a to a next module, 310 b, which further re-drives the memorybus 315 to module positions denoted as 310 n. Each module 310 a includesa DRAM 311 a and a buffer 320 a. The memory bus 315 may be described ashaving a daisy chain structure with each bus being point-to-point innature.

A variety of factors including faulty components and inadequate designtolerances may result in errors in the data being processed by a memorysubsystem. Errors may also occur during data transmission due to “noise”in the communication channel (e.g., the bus 315). As a result of theseerrors, one or more bits, which may be represented as X, which are to betransmitted within the system, are corrupted so as to be received as“/X” (i.e., the logical complement of the value of X). In order toprotect against such errors, the data bits may be coded via an errorcorrecting code (ECC) in such a way that the errors may be detected andpossibly corrected by special ECC logic circuits. A typical ECCimplementation appends a number of check bits to each data word. Theappended check bits are used by the ECC logic circuits to detect errorswithin the data word. By appending bits (e.g., parity bits) to the dataword, each bit corresponding to a subset of data bits within the dataword, the parity concepts may be expanded to provide the detection ofmultiple bit errors or to determine the location of single or multiplebit errors. Once a data bit error is located, a logic circuit may beutilized to correct the located erroneous bit, thereby providing singleerror correction (SEC). Many SEC codes have the ability to detect doubleerrors and are thus termed SEC double error detecting (SEC-DED) codes.

FIG. 8 represents a typical parallel bus ECC structure that transfers acomplete ECC word in a single cycle. The structure depicted in FIG. 8 isconsistent with the teachings in U.S. Pat. No. 6,044,483 to Chen et al.,of common assignment herewith. FIG. 8 depicts an 88/72 ECC for computersystems having an eight bit per chip memory configuration. The lineslabeled “Wire 0” through “Wire 72” each represent a wire on the memorybus 315 with seventy-two wires. For a memory subsystem with an eight bitper chip memory configuration, sixty-four bits of data and eight ECCbits are transferred every cycle. The ECC word is transferred entirelyin one cycle, and a SEC-DED code may be utilized to correct any singlebit failure anywhere in the ECC word, including a hard wire or bitlanefailure. In the case of a hard wire or bitlane failure, every transferhas the same bitlane in error with the ECC correcting it for eachtransfer.

FIG. 9 depicts a typical manner of defining symbol ECCs for use in faultdetection and correction in a memory subsystem. FIG. 8 is consistentwith the teachings of U.S. Pat. No. 6,044,483. As shown in FIG. 9, thesymbols are four bits in length and the symbols are defined acrossbitlanes. As is known in the art, a symbol refers to a mathematicalderivation of ECC and corresponds to a group of bits that the ECC isable to correct either individually or as a group. Referring to FIG. 9,assuming that data bits one through four are sourced from the samememory chip, respectively, data errors located by “symbol 1” can belocalized to a particular memory chip (e.g., a DRAM).

Busses that are protected by ECC are typically run as single transferbusses with a SEC-DED code. In other words, any single bitlane failureis corrected by the SEC code because the ECC word is completelytransmitted in one cycle (or shot or transfer). Thus, if a wire,contact, or bitlane is faulty, it would be a faulty bit in everytransfer, and the SEC ECC will correct the error each cycle.

Defining symbols across bitlanes may be used to effectively isolateerrors to memory chips when a relatively wide parallel ECC structure isimplemented and a complete ECC word is transferred in a single cycle.However, defining symbols across bitlanes may not be effective inisolating errors to a particular memory chip or bus wire when arelatively narrow parallel interface is implemented with the ECC word(made up of data bits and ECC bits) being delivered in packets overmultiple cycles.

BRIEF SUMMARY OF THE INVENTION

Exemplary embodiments of the present invention include a memorysubsystem with a memory bus and a memory assembly. The memory busincludes multiple bitlanes. The memory assembly is in communication withthe memory bus and includes instructions (circuitry and/or software) forreceiving an error code correction (ECC) word in multiple packets viathe memory bus. The ECC word includes data bits and ECC bits arrangedinto multiple multi-bit ECC symbols. Each of the ECC symbols isassociated with one of the bitlanes on the memory bus. The memoryassembly also includes instructions for utilizing one of the ECC symbolsto perform error detection and correction for the bits in the ECC wordreceived via the bitlane associated with the ECC symbol.

Additional exemplary embodiments include a memory subsystem with amemory bus and a memory assembly. The memory bus includes multiplebitlanes. The memory assembly is in communication with the memory busand includes instructions (circuitry and/or software) for creating anECC word. The ECC word includes data bits and ECC bits arranged intomultiple multi-bit ECC symbols. Each of the ECC symbols is associatedwith one of the bitlanes on the memory bus. The memory assembly alsoincludes instructions for transmitting the ECC word in multiple packetsvia the memory bus.

Further exemplary embodiments include a method for providing errordetection and correction. The method includes receiving an ECC word at amemory assembly in multiple packets via a memory bus. The ECC wordincludes data bits and ECC bits arranged into multiple ECC symbols. Eachof the ECC symbols is associated with one bitlane on the memory bus. Themethod further includes utilizing one of the ECC symbols to performerror detection and correction to bits in the ECC word received via thebitlane associated with the symbol.

Still further exemplary embodiments include a storage medium encodedwith machine readable computer program code for providing faultdetection and correction in a memory subsystem. The storage mediumincludes instructions for causing a computer to implement a method. Themethod includes receiving an ECC word at a memory assembly in multiplepackets via a memory bus. The ECC word includes data bits and ECC bitsarranged into multiple ECC symbols. Each of the ECC symbols isassociated with one bitlane on the memory bus. The method furtherincludes utilizing one of the ECC symbols to perform error detection andcorrection to bits in the ECC word received via the bitlane associatedwith the symbol.

An additional exemplary embodiment of the present invention includes acommunication system with a bus containing multiple bitlanes and adevice in communication with the bus. The device includes instructionsfor receiving an ECC word in multiple packets via the bus. The ECC wordincludes data bits and ECC bits arranged into multiple multi-bit ECCsymbols with each of the ECC symbols being associated with one of thebitlanes on the bus. One of the ECC symbols is utilized to perform errordetection and correction for the bits in the ECC word received via thebitlane and associated with the ECC symbol. The number of ECC bits isgreater than a second number of ECC bits in a SEC/DED implementation ofequivalent bitlane dimension. In addition, the number of bitlanesutilized to transfer the ECC bits is less than a second number ofbitlanes in an SEC/DED implementation of equivalent bitlane dimension.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings wherein like elements are numbered alikein the several FIGURES:

FIG. 1 depicts a prior art memory controller connected to two bufferedmemory assemblies via separate point-to-point links;

FIG. 2 depicts a prior art synchronous memory module with a bufferdevice;

FIG. 3 depicts a prior art memory subsystem using registered DIMMs;

FIG. 4 depicts a prior art memory subsystem with point-to-pointchannels, registered DIMMs, and a 2:1 bus speed multiplier;

FIG. 5 depicts a prior art memory structure that utilizes a multidropmemory ‘stub’ bus;

FIG. 6 depicts a prior art daisy chain structure in a multipointcommunication structure that would otherwise require multiple ports;

FIG. 7 depicts a prior art daisy chain connection between a memorycontroller and memory modules;

FIG. 8 represents a prior art parallel bus ECC structure that transfersa complete ECC word in a single cycle;

FIG. 9 depicts a prior art manner of defining symbol ECCs, for use infault detection and correction in a memory subsystem;

FIG. 10 depicts a cascaded memory structure that is utilized byexemplary embodiments of the present invention;

FIG. 11 depicts a memory structure with cascaded memory modules andunidirectional busses that is utilized by exemplary embodiments of thepresent invention;

FIG. 12 depicts a buffered module wiring system that is utilized byexemplary embodiments of the present invention;

FIG. 13 depicts a downstream frame format that is utilized by exemplaryembodiments of the present invention;

FIG. 14 depicts an upstream frame format that is utilized by exemplaryembodiments of the present invention;

FIG. 15 is a table with symbols defined across bitlanes, using theupstream frame format depicted in FIG. 14 as an example; and

FIG. 16 is a table with symbols defined within bitlanes, using theupstream frame format depicted in FIG. 14 as an example.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Exemplary embodiments of the present invention utilize a symbol ECCinterleaved such that the symbol orientation is along the bitlanes. Arelatively narrow parallel interface is utilized with data that isdelivered in packets over several transfers. Thus, the ECC word istwo-dimensional in that the data bits and ECC bits are delivered inmultiple cycles over a parallel interface. In order to provideprotection for both random noise related soft errors on the bus, as wellas systematic hard errors (errors such as a contact or wire failurealong the bitlane), a new ECC scheme is employed. In addition, thecapability of expanding the code to a higher order of robustness, forexample going from a single symbol correcting code to a double symbolcorrecting code by using a spare wire(s), is available in this scheme.Exemplary embodiments of the present invention are contrasted against adetect/retry scheme where once an error is detected the data is re-sent.In exemplary embodiments of the present invention, errors are correctedon the fly, and in the event of a spare being deployed, the full ECCcode is still available. Exemplary embodiments of the present inventionprovide an 88/72 ECC code that will detect all errors on a four bitsymbol, correct all four bit symbol errors and detect all double fourbit symbol errors (e.g., single 4-bit error correcting/double 4-biterror detecting (S4EC/D4ED).

SEC/DED=Single Error Correcting/Double Error Detecting

S4EC/D4ED=Single 4-bit Error Correcting/Double 4-bit Error Detecting

In an exemplary embodiment of the present invention, fault correctionand detection is provided by a high speed and high reliability memorysubsystem architecture and interconnect structure that includessingle-ended point-to-point interconnections between any two subsystemcomponents. The memory subsystem further includes a memory controlfunction, one or more memory modules, one or more high speed bussesoperating at a four-to-one speed ratio relative to a DRAM data rate anda bus-to-bus converter chip on each of one or more cascaded modules toconvert the high speed bus(ses) into the conventional double data rate(DDR) memory interface. The memory modules operate as slave devices tothe memory controller, responding to commands in a deterministic manner,but do not self-initiate unplanned bus activity, except in cases whereoperational errors are reported in a real-time manner. Memory modulescan be added to the cascaded bus with each module assigned an address topermit unique selection of each module on the cascaded bus. Exemplaryembodiments of the present invention include a packetized multi-transferinterface which utilizes an innovative communication protocol to permitmemory operation to occur on a reduced pincount, whereby address,command and data is transferred between the components on the cascadedbus over multiple cycles, and are reconstructed and errors correctedprior to being used by the intended recipient.

FIG. 10 depicts a cascaded memory structure that may be utilized byexemplary embodiments of the present invention when buffered memorymodules 1006 (e.g., the buffer device is included within the memorymodule 1006) are in communication with a memory controller 1002. Thismemory structure includes the memory controller 1002 in communicationwith one or more memory modules 1006 via a high speed point-to-point bus1004. Each bus 1004 in the exemplary embodiment depicted in FIG. 10includes approximately fifty high speed wires for the transfer ofaddress, command, data and clocks. By using point-to-point busses asdescribed in the aforementioned prior art, it is possible to optimizethe bus design to permit significantly increased data rates, as well asto reduce the bus pincount by transferring data over multiple cycles.Whereas FIG. 4 depicts a memory subsystem with a two to one ratiobetween the data rate on any one of the busses connecting the memorycontroller to one of the bus converters (e.g., to 1,066 Mb/s per pin)versus any one of the busses between the bus converter and one or morememory modules (e.g., to 533 Mb/s per pin), an exemplary embodiment ofthe present invention, as depicted in FIG. 10, provides a four to onebus speed ratio to maximize bus efficiency and minimize pincount.

Although point-to-point interconnects permit higher data rates, overallmemory subsystem efficiency must be achieved by maintaining a reasonablenumber of memory modules 1006 and memory devices per channel(historically four memory modules with four to thirty-six chips permemory module, but as high as eight memory modules per channel and asfew as one memory module per channel). Using a point-to-point busnecessitates a bus re-drive function on each memory module to permitmemory modules to be cascaded such that each memory module isinterconnected to other memory modules, as well as to the memorycontroller 1002.

FIG. 11 depicts a memory structure with cascaded memory modules andunidirectional busses that is utilized by exemplary embodiments of thepresent invention if all of the memory modules 1006 are buffered memorymodules 1006. One of the functions provided by the memory modules 1006in the cascade structure is a re-drive function to send signals on thememory bus to other memory modules 1006 or to the memory controller1002. FIG. 11 includes the memory controller 1002 and four memorymodules 1006 a, 1006 b, 1006 c and 1006 d, on each of two memory busses(a downstream memory bus 1104 and an upstream memory bus 1102),connected to the memory controller 1002 in either a direct or cascadedmanner. Memory module 1006 a is connected to the memory controller 1002in a direct manner. Memory modules 1006 b, 1006 c and 1006 d areconnected to the memory controller 1002 in a cascaded manner.

An exemplary embodiment of the present invention includes twouni-directional busses between the memory controller 1002 and memorymodule 1006 a (“DIMM #1”), as well as between each successive memorymodule 1006 b-d (“DIMM #2”, “DIMM #3” and “DIMM #4”) in the cascadedmemory structure. The downstream memory bus 1104 is comprised oftwenty-two single-ended signals and a differential clock pair. Thedownstream memory bus 1104 is used to transfer address, control, dataand error code correction (ECC) bits downstream from the memorycontroller 1002, over several clock cycles, to one or more of the memorymodules 1006 installed on the cascaded memory channel. The upstreammemory bus 1102 is comprised of twenty-three single-ended signals and adifferential clock pair, and is used to transfer bus-level data and ECCbits upstream from the sourcing memory module 1006 to the memorycontroller 1002. Using this memory structure, and a four to one datarate multiplier between the DRAM data rate (e.g., 400 to 800 Mb/s perpin) and the unidirectional memory bus data rate (e.g., 1.6 to 3.2 Gb/sper pin), the memory controller 1002 signal pincount, per memorychannel, is reduced from approximately one hundred and twenty pins toabout fifty pins.

FIG. 12 depicts a buffered module wiring system that is utilized byexemplary embodiments of the present invention. FIG. 12 is a pictorialrepresentation of a memory module with shaded arrows representing theprimary signal flows. The signal flows include the upstream memory bus1102, the downstream memory bus 1104, memory device address and commandbusses 1210 and 1206, and memory device data busses 1204 and 1208. In anexemplary embodiment of the present invention, a buffer device 1202,also referred to as a memory interface chip provides two copies of theaddress and command signals to the synchronous DRAMs (SDRAMs) 1204 withthe right memory device address and command bus 1206 exiting from theright side of the buffer device 1202 for the SDRAMs 1204 located to theright side and behind the buffer device 1202 on the right. The leftmemory device address and command bus 1210 exits from the left side ofthe buffer device 1202 and connects to the SDRAMs 1204 to the left sideand behind the buffer device 1202 on the left. Similarly, the data bitsintended for SDRAMs 1204 to the right of the buffer device 1202 exitfrom the right of the buffer module 1202 on a right memory device databus 1208. The data bits intended for the left side of the buffer device1202 exit from the left of the buffer device 1202 on the left memorydevice data bus 1204. The high speed upstream memory bus 1102 anddownstream memory bus 1104 exit from the lower portion of the bufferdevice 1202, and connect to a memory controller or other memory moduleseither upstream or downstream of this memory module 1006, depending onthe application. The buffer device 1202 receives signals that are fourtimes the memory module data rate and converts them into signals at thememory module data rate.

FIG. 13 depicts a downstream frame format that is utilized by exemplaryembodiments of the present invention to transfer information downstreamfrom the memory controller 1002 to the memory modules 1006. In anexemplary embodiment of the present invention, the downstream frameconsists of eight transfers with each transfer including twenty-twosignals and a differential clock (twenty-four wires total). The framefurther consists of eight command wires (c0 through c7) 1308, nine datawires (di0 through di8) 1306, four bus error correction code (ECC) wires(ecc0 through ecc3) 1304 and a spare wire (spare) 1302. The seventy-twodata bits are shown in FIG. 13 as bits di0 through di8, and consist ofnine wires with eight transfers on each wire for each frame. Inexemplary embodiments of the present invention, the frame formatdepicted in FIG. 13 may be utilized to deliver one or two memorycommands plus seventy-two bits of write data per memory clock cycle. Thenumbering of each data bit, as well as for other bits, is based on thewire used, as well as the specific transfer. D34 refers to data bit 3(of bits 0 through 8) and transfer 4 (of transfer 0 through 7). Thecommand bit field is shown as c0 through c7, and consists of sixty-fourbits of information provided to the module over eight transfers.

The ECC bit field (ecc0 through ecc3) consists of thirty-two bitpositions over eight transfers but is actually formatted in groups ofsixteen bits. Each sixteen bit packet consists of four transfers overeach of the four wires and provides the bus level fault detection andcorrection across each group of four bus transfers. The bus level errorcorrecting code fault detection and correction is performed by a logicblock that includes instructions to carry out the detection andcorrection. The spare bit position may be used to logically replace anyof the twenty-one wires, also defined as bitlanes, used to transfer bitsin the command, data and ECC fields should a failure occur in one of thebitlanes that results in exceeding a system-assigned failure thresholdlimit. The spare wire may be utilized to replace a failing segmentbetween any two directly connected assemblies (i.e., between the memorycontroller 1002 and the memory module 1006 a, or between any two memorymodules 1006 a-d), to replace a wire due to events, such as a wirefailure, a connector failure, a solder interconnect failure, a driverfailure and/or a receiver failure. Out of the one hundred andseventy-six possible bit positions, one hundred and sixty-eight areavailable for the transfer of information to the memory module 1006, andof those one hundred and sixty-eight bit positions, thirty-two bitpositions are further assigned to providing ECC protection on the bustransfers themselves, thereby allowing a total of one hundred andthirty-six bit positions to be used for the transfer of information tothe memory module 1006. The frame format depicted in FIG. 13 isapplicable to incoming signals to the memory module 1006 from thedirection of the memory controller, as well as the outgoing signals toany downstream memory modules 1006.

FIG. 14 depicts an upstream frame format that is utilized by exemplaryembodiments of the present invention to transfer information upstreamfrom the memory module 1006 to either the memory controller 1002 or anupstream memory module 1006. In an exemplary embodiment of the presentinvention, the upstream frame consists of eight transfers with eachtransfer including twenty-three signals and a differential clock(twenty-five wires total). The frame further consists of eighteen datawires (do0 through do17) 1406, four bus ECC wires (ecc0 through ecc3)1404 and a spare wire (spare) 1402. In exemplary embodiments of thepresent invention, the frame format depicted in FIG. 14 may be utilizedto deliver one hundred and forty-four read data bits per memory clockcycle. The numbering of each data bit, as well as for other bits, isbased on the wire used, as well as the specific transfer. D34 refers todata bit 3 (of bits 0 through 17) and transfer 4 (of transfer 0 through7).

The ECC bit field (ecc0 through ecc3) consists of thirty-two bitpositions over eight transfers but is actually formatted in groups ofsixteen bits. Each sixteen bit packet consists of four transfers overeach of the four wires with error correction being performed every fourtransfers. The spare wire position may be used to logically replace anyof the twenty-two wires used to transfer bits in the data and ECC fieldsshould a failure occur in one of these wires that is consistent innature. A failure may be considered to be consistent in nature if itexceeds a system dependent threshold value (e.g., number of times thefailure is detected). Single bitlane failures may be corrected on thefly by the bus level ECC, while a system service element, such as aservice processor, may decide to spare out a failing segment to repairhard (e.g., periodic, repeating and continuous) failures that may occurduring system operation.

FIG. 15 is a table with symbols defined across bitlanes, using theupstream frame format depicted in FIG. 14 as an example. Each column1506 represents the bits of a symbol (i.e., “bit 1”, “bit 2”, “bit 3”,and “bit 4”). Each row represents a symbol (i.e., “symbol 1”, “symbol2”, etc.). Bit 1 of symbol 1 contains data bit D00 from FIG. 14, bit 2of symbol 1 contains data bit D10 from FIG. 14, bit 3 contains data bitD20 from FIG. 14 and bit 4 contains data bit D30 from FIG. 14. All ofthe data bits contained in symbol 1 are transferred during the firsttransfer within the frame, or transfer 0. As described previously, theECC is checked for every four transfers in a frame, or for every halfframe. The ECC word, including eighty-eight bits (the spare bitlane isincluded in place of a failing bitlane only if it is being utilized), istransferred over multiple cycles (e.g., four) such that a hard fail inany bitlane or wire will result in a multiple-bit error. A typicalsingle symbol correcting/double symbol detecting (SSC/DSD) ECC schemewill not be effective in correcting hard bitlane failures for the symbolscheme depicted in FIG. 15 because the bits within the symbol are notbeing sent on the same bitlane. Also, a SSC/DSD code would not beeffective either because symbols are generally distributed amongadjacent bits of a bus. This approach to forming symbols acrossbitlanes, as applied to the upstream format shown in FIG. 14, is shownin FIG. 15.

FIG. 16 is a table with symbols defined along the bitlanes, inaccordance with exemplary embodiments of the present invention, usingthe upstream frame format depicted in FIG. 14 as an example. The firstfour columns 1604 in FIG. 16 represent the bits of a symbol (i.e., “bit1”, “bit 2”, “bit 3”, and “bit 4”). The last column 1606 contains thebitlane being covered by the symbol. Each row represents a symbol (i.e.,“symbol 1”, “symbol 2”, etc.). Bit 1 of symbol 1 contains data bit D00from FIG. 14, bit 2 of symbol 1 contains data bit D01 from FIG. 14, bit3 contains data bit D02 from FIG. 14 and bit 4 contains data bit D03from FIG. 14. Symbol 1 covers bitlane 5 in FIG. 14. Bit 1 wastransferred during transfer zero, bit 2 during transfer one, bit 3during transfer two and bit 4 during transfer three. Interleaving thedata bits in the ECC code such that the ECC symbol is defined along thebitlanes allows the data bits to be protected along the bitlanedimension rather than in the databit dimension depicted in FIG. 15.Again, the ECC is checked every four transfers in a frame, or for everyhalf frame. The ECC word, including eighty-eight bits (the spare bitlaneis included in place of a failing bitlane only if it is being utilized),is transferred over multiple cycles (e.g., four). The ECC word includesboth data bits and ECC bits. Data bits may include, but are not limitedto, command bits, write data bits and read data bits. In addition, ECCbits may be treated as data bits with the ECC processing describedherein being applied to the ECC bits.

The symbols in FIG. 16 are defined to span a single bitlane, and if anyof the hardware that is associated with any of the bitlanes fails, thefailure will affect all four of the data bits that are transferred viathe failing bitlane. Orienting the symbol along the bitlanes allows asingle and/or multiple errors caused by a failing bitlane to be detectedand corrected.

A further advantage of defining symbols along bitlanes is that ituncouples the requirement to have unique data/address/command formatwith the ECC word. Exemplary embodiments of the present invention allowdata bits one through seventy-two to be any combination of data, addressor command information.

Another benefit to defining symbols along bitlanes is that it isflexible and modular in that an additional bitlane can be added to thebus for the purposes of additional data, additional robustness, or evenas a spare. Because the additional bitlane is dedicated to the addedfunction, no further structural changes are required to allow thebenefits of the added wire to accrue (either in more data capacity,spare capability, or a more robust ECC).

In addition, the scheme of defining symbols along bitlanes can beexpanded to include an arbitrary number of transfers. This will causethe ECC symbol size to commensurately grow, but conceptually andstructurally, this idea can be extended in the transfer dimensionwithout any loss of coverage.

In general, the narrower the bus, the greater percentage of signals thatmust be checkbits to provide the same level of protection. For classicHamming codes, or traditional SEC/DED bus ECC, the following tableillustrates this point: Maximum Data Signals Checkbit Signals PercentOverhead 4 4 50% 11 5 31% 26 6 19% 57 7 11% 120 8  6% 247 9  4%This can be compared to exemplary embodiments of the present inventionincluding symbol oriented bus ECC where 4 checkbits are utilized for upto 253 data signals with a resulting overhead of 1.6%. Therefore, sixcheckbits could have been utilized to support 17 or 18 databits if thetraditional SEC/DED had been utilized.

One drawback to using the traditional SEC/DED for read and write databusses that are daisy-chained through the support ASIC is that eachadditional checkbit gets multiplied by four in terms of ASIC pincount(i.e., 2 additional checkbits for each bus equals 8 additional ASICpins). Another drawback to using the traditional SEC/DED is that theprobability that a code will detect random, multibit errors is inverselyproportional to the expression “2 raised to the nth power”, where n isthe number of checkbits. This means that a (24,18) SEC/DED code with 6checkbits has a figure of merit of 1/(2⁶)=0.016. This is contrasted toexemplary embodiments of the present invention described herein thathave a figure of merit of 1/(2¹⁶)=0.000015, which is about 100 timesbetter. A further advantage of exemplary embodiments of the presentinvention over the traditional SEC/DED approach is that the currentdesign point described herein of 18 data bitlanes can be expanded tosupport up to 253 bitlanes without adding new checkbits. In contrast,the traditional SEC/DED approach would require the addition of checkbitssuch that even adding 9 additional data bitlanes (for a total of 27)would require an additional checkbit.

As described above, the embodiments of the invention may be embodied inthe form of computer-implemented processes and apparatuses forpracticing those processes. Embodiments of the invention may also beembodied in the form of computer program code containing instructionsembodied in tangible media, such as floppy diskettes, CD-ROMs, harddrives, or any other computer-readable storage medium, wherein, when thecomputer program code is loaded into and executed by a computer, thecomputer becomes an apparatus for practicing the invention. The presentinvention can also be embodied in the form of computer program code, forexample, whether stored in a storage medium, loaded into and/or executedby a computer, or transmitted over some transmission medium, such asover electrical wiring or cabling, through fiber optics, or viaelectromagnetic radiation, wherein, when the computer program code isloaded into and executed by a computer, the computer becomes anapparatus for practicing the invention. When implemented on ageneral-purpose microprocessor, the computer program code segmentsconfigure the microprocessor to create specific logic circuits.

While the invention has been described with reference to exemplaryembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the invention. Inaddition, many modifications may be made to adapt a particular situationor material to the teachings of the invention without departing from theessential scope thereof. Therefore, it is intended that the inventionnot be limited to the particular embodiment disclosed as the best modecontemplated for carrying out this invention, but that the inventionwill include all embodiments falling within the scope of the appendedclaims. Moreover, the use of the terms first, second, etc. do not denoteany order or importance, but rather the terms first, second, etc. areused to distinguish one element from another.

1. A memory subsystem, the memory subsystem comprising: a memory buscomprising multiple bitlanes; and a memory assembly in communicationwith the memory bus, the memory assembly including instructions for:receiving an error code correction (ECC) word in multiple packets viathe memory bus, the ECC word including data bits and ECC bits arrangedinto multiple multi-bit ECC symbols, wherein each of the ECC symbols isassociated with one of the bitlanes on the memory bus; and utilizing oneof the ECC symbols to perform error detection and correction for thebits in the ECC word received via the bitlane associated with the ECCsymbol.
 2. The memory subsystem of claim 1 wherein the memory buscomprises twenty-two bitlanes.
 3. The memory subsystem of claim 2wherein eighteen of the twenty-two bitlanes contain the data bits andfour of the twenty-two bitlanes contain the ECC bits.
 4. The memorysubsystem of claim 2 wherein the memory bus further comprises a sparebitlane.
 5. The memory subsystem of claim 1 wherein the receiving an ECCword requires four packets to be transferred via the memory bus.
 6. Thememory subsystem of claim 1 wherein the symbol includes four bits. 7.The memory subsystem of claim 1 wherein the memory assembly furtherincludes instructions for creating a new ECC word and transmitting thenew ECC word.
 8. The memory subsystem of claim 1 wherein the memory busand the memory assembly are part of a cascaded interconnect memorysystem.
 9. The memory subsystem of claim 1 wherein the memory assemblyis a memory controller or a memory module.
 10. The memory subsystem ofclaim 1 wherein the memory bus is a downstream bus.
 11. The memorysubsystem of claim 1 wherein the memory bus is an upstream bus.
 12. Thememory subsystem of claim 1 wherein the ECC bits are also data bits. 13.A memory subsystem, the memory subsystem comprising: a memory buscomprising multiple bitlanes; and a memory assembly in communicationwith the memory bus, the memory assembly including instructions for:creating an error code correction (ECC) word, the ECC word includingdata bits and ECC bits arranged into multiple multi-bit ECC symbols,wherein each of the ECC symbols is associated with one of the bitlaneson the memory bus; and transmitting the ECC word in multiple packets viathe memory bus.
 14. A method for providing error detection andcorrection, the method comprising: receiving an error code correction(ECC) word at a memory assembly in multiple packets via a memory bus,the ECC word including data bits and ECC bits arranged into multiple ECCsymbols, each of the ECC symbols associated with one bitlane on thememory bus; and utilizing one of the ECC symbols to perform errordetection and correction to bits in the ECC word received via thebitlane associated with the symbol.
 15. The method of claim 14 whereinthe memory assembly is a memory controller or a memory module.
 16. Themethod of claim 14 wherein the memory bus comprises twenty-two bitlanes.17. The method of claim 16 wherein eighteen of the twenty-two bitlanescontain data bits and four of the twenty-two bitlanes contain the ECCbits.
 18. The method of claim 14 wherein the receiving an ECC wordrequires four packets to be transferred via the memory bus.
 19. Themethod of claim 14 wherein each of the symbols includes four bits. 20.The method of claim 14 wherein the memory assembly further includesinstructions for creating a new ECC word with symbols and transmittingthe new ECC word.
 21. The method of claim 14 wherein the memory bus andthe memory assembly are part of a cascaded interconnect memory system.22. The method of claim 14 wherein the memory bus is a downstream bus.23. The method of claim 14 wherein the memory bus is an upstream bus.24. The method of claim 14 wherein each ECC symbol is four bits and theECC word includes an 88/72 ECC code that will detect errors in thesymbol, correct errors in the symbol and detect double symbol errors.25. The method of claim 14 wherein the ECC bits are also data bits. 26.A storage medium encoded with machine readable computer program code forproviding fault detection and correction in a memory subsystem, thestorage medium including instructions for causing a computer toimplement a method comprising: receiving an error code correction (ECC)word at a memory assembly in multiple packets via a memory bus, the ECCword including data bits and ECC bits arranged into multiple ECCsymbols, each of the ECC symbols associated with one bitlane on thememory bus; and utilizing one of the ECC symbols to perform errordetection and correction to bits in the ECC word received via thebitlane associated with the symbol.
 27. A communication systemcomprising: a bus comprising multiple bitlanes; and a device incommunication with the bus, the device including instructions for:receiving an error code correction (ECC) word in multiple packets viathe bus, the ECC word including data bits and ECC bits arranged intomultiple multi-bit ECC symbols, wherein each of the ECC symbols isassociated with one of the bitlanes on the bus; and utilizing one of theECC symbols to perform error detection and correction for the bits inthe ECC word received via the bitlane and associated with the ECCsymbol, wherein the number of ECC bits is greater than a second numberof ECC bits in a single error correcting/double error detecting(SEC/DED) implementation of equivalent bitlane dimension and the numberof said bitlanes utilized to transfer the ECC bits is less than a secondnumber of bitlanes in an SEC/DED implementation of equivalent bitlanedimension.
 28. The system of claim 27 wherein the communication systemis a memory subsystem, the bus is a memory bus and the device is amemory device.
 29. The system of claim 27 wherein the number of saidbitlanes is between 16 and 254 and the number of ECC bits per packet isfour.